Global mapping of epidemic risk assessment toolkits: A scoping review for COVID-19 and future epidemics preparedness implications

Preparedness and responses to infectious disease epidemics and pandemics require the understanding of communities’ and multisectoral systems’ characteristics with regards to diseases transmission and population’s vulnerabilities. This study aimed to summarize measurement profiles of existing risk assessment toolkits to inform COVID-19 control at global and national levels. An online search in different databases and online sources was performed to identify all epidemic risk and vulnerability assessment instruments. Medline/PubMed, Web of Science databases, and websites of public health organizations were used for the searching process. Of 14 toolkits, levels of setting were mostly at the global or nation level. Components such as Governance and Legislation, Financing, Health Service Provision, and Human Resources are key domains in almost all toolkits. Some important issues for disease detection and surveillance, such as laboratory or capacity of the community for disease control, were not adequately addressed in several toolkits. Limited studies were found that validated the toolkits. Only five toolkits were used in COVID-19 studies. This study provides a summary of risk assessment toolkits to inform epidemic responses. We call for global and national efforts in developing more contextualized and responsive epidemic risk assessment scales incorporating specific-disease and -country factors to inform operational decisions making and strengthen countries’ capacities in epidemic responses.


Introduction
Novel coronavirus disease 2019 (COVID -19) has been officially recognized as a global pandemic by the World Health Organization, with more than 523 million positive cases and 6.27 million deaths reported in approximately 220 countries and territories until 19 May 2022 [1]. This widespread transmission is understandable since only half of the states had sufficient preparations for this health emergency, according to a global analysis in 2020 [2]. From 2020 to 2021, the number of cases and deaths still remarkably increase although many efforts have been performed to prevent and control the pandemic in all nations, from accelerating the vaccination coverage, strict quarantine, and preventive measures such as social distancing, contact tracing, face masks mandatory [1]. The presence of the Delta variant and other potential variants have made the governments of countries realize that the "Zero Covid" strategy is completely impossible (except for China) and need to develop strategies for living with COVID-19 in the future [3].
There is no doubt that residents in all countries, regardless of economic conditions, are currently more vulnerable to infectious disease outbreaks [4]. The number of infectious disease epidemics over the last 20 years, at both local and global levels, is far beyond the number of epidemics in the whole last century [5], resulting in remarkable health and economic losses in affected nations [4]. This explosion is attributable to substantial increases in international connectivity, population density, and human-wild animal interactions, as well as amplified by alterations of ecological factors such as climate change or rapid urbanization [6][7][8]. Notably, in addition to existing periodical infectious diseases (for example, dengue, malaria, or influenza), a diversity of novel disease epidemics with high morbidity and mortality rates have been recorded such as severe acute respiratory syndrome-SARS, H1N1, Zika, Middle East Respiratory Syndrome-MERS or, most recently, COVID-19 [9,10]. The emergence of all pathogens is unpredictable, but they began with some local cases then spread out to become international crises due to global travel and trade [11]. Therefore, albeit the unpredictability of novel epidemic agents, preparedness in each country is critically important to respond to localized outbreaks, prevent the spread, and mitigate the epidemic's burden [12].
Preparedness and responses to infectious disease epidemics and pandemics require the understanding of communities' and systems' characteristics with regards to diseases transmission and population's vulnerabilities [4,12]. Since the International Health Regulations (IHR) was issued (2005) and came into force (2007) [13], there have been substantial efforts to quantify the total risks and assess vulnerabilities of different populations at national, regional, and global levels. For instance, in 2005, Joint External Evaluation Tool (JEE) was developed that adopted the IHR regulations to externally assess the country's capacity to detect and respond to the public health risks [14]. In 2007, the State Parties Self-Assessment Annual Reporting Tool (SPAR) was also constructed to measure the progress in acquiring IHR targets [15]. The most recent initiatives for risk and vulnerability assessment were the Epidemic Preparedness Index [12], the Health Vulnerability Index for Disaster Risk Reduction [16], the Global Health Security Index (GHSI) [17], and the EpiRisk Tool [18], which use open-source data for national level-gap analysis.
Examples of the use of these assessment toolkits for epidemic preparation have been described in the literature. Espinal et al. analyzed IHR's core capacities to inform the gaps of Latin America and the Caribbean countries before Ebola outbreaks and suggested that countries should strengthen their capacities and monitoring approaches [19]. Glynn et al. assessed the preparedness of Ireland against Ebola and Zika epidemics, which had occurred in West Africa in 2007-2008, and found that the country had a good preparation before these epidemics [20]. Some instruments have been applied to evaluate national preparedness for COVID-19 outbreaks. For example, Craig et al. analyzed GHSI data of 112 countries and showed that 54/ 112 countries had scored lower than the global average [21]. The authors also indicated that all Pacific Island countries and territories belonged to the lowest preparedness group for COVID-19 [21]. Another study using SPAR by Kandel et al. revealed that among 182 countries, 24% (44 countries) did not have any effective mechanisms to enable responses to COVID-19 [2]. Wong et al. showed that a higher IHR score was negatively associated with the number of new COVID-19-related cases and deaths [22].
To date, limited studies attempted to have the consensus in accordance with the necessary elements that should be included in the epidemic risk assessment. This study summarized the developmental history, profiles, and applications of existing toolkits for evaluating global risks and vulnerabilities of infectious diseases. Second, it analyzed gaps in evidence and implications of these existing measurements for COVID-19 preparedness and responses at national and global levels.

Search strategy
In this study, we employed two searching strategies for identifying the epidemic risk assessment toolkits: 1) Online search in electronic databases to identify scientific peer-reviewed articles; and 2) Online search in public health organization websites to identify grey literature. First, we conducted the online search in Medline/PubMed and Web of Science databases to identify the peer-reviewed articles published from January 1 st, 2000 to June 30 th, 2021. The searches strings and results are presented in Table 1. We combined all searches strings by using the Boolean operator "AND". Second, we searched grey literature on websites of the following organizations: World Health Organization (WHO), the United States Center for Disease Control and Prevention (CDC), the United States Agency for International Development (USAID), European Commission, and the European CDC. We also sought in the references of these selected documents additional eligible publications. The searching process was performed in April 2020 and updated in July 2021.

Selection criteria
All articles or documents related to the development and use of epidemic risk and vulnerability assessment instruments were included. Other eligibility criteria included: 1) Being published in English; 2) Published from January 1 st 2000 to June 30 th 2021; 3) Covering international, national or subnational assessment; 4) Presence of checklists, indicators or scales for assessing national epidemic risk and vulnerability. For those published from 2020 to 2021, we also sought publications that used these tools in COVID-19 topics to examine the application of these tools. We excluded papers covering instruments that only focused on specific diseases, hazards which were not infectious diseases or natural reasons (for instance, bioterrorismrelated epidemic) such as the World Health Organization Measles Programmatic Risk Assessment Tool, the WHO human health risk assessment toolkit for chemical hazards, or Risk assessment guidelines for diseases transmitted on aircraft, or Joint European Pandemic Preparedness Self Assessment Indicators (focusing on influenza), which might not be widely applied for other conditions. Civil emergency assessments were excluded if they did not cover epidemic risk assessment or preparedness or response. We also excluded papers that were 1) narrative review, systematic review or meta-analysis; 2) abstract, study protocol, conference paper, conference proceedings, news, letters and others that were not scientific articles or organization's reports or guidelines (if they mentioned epidemic risk assessment scales).

Data extraction and synthesis
Extracted information included the name of instruments, year of the first publication, origin, sources of data for assessment, number of domains, name of domains, number of items, and score range. We identified whether an instrument was correlated with each other (concurrent validity) by extracting the data of correlation measures (for example, Pearson or Spearman correlation coefficient, regression coefficient, odds ratio, relative risk, or hazard ratio). Information about the application of these tools in the COVID-19 pandemic was extracted in publications reporting these topics of interest. Two independent researchers performed the data extraction. Disagreements were solved by the third senior researcher. Table 2 described the evaluation framework used to compare different assessment tools, which were adapted from previous reviews [23][24][25]. Clarity of measurement Extend to which the manners to measure the items/indicators/parameters were described in the tool.
Validity of measurement Extend to which the tool's score was correlated with other tools (criterion validity), correlated with other epidemic outcomes and responses (construct validity), and correlated with COVID-19 pandemic outcomes (predictive validity)

Scope
Extend to which the data was applied for, for example, international, national, or community level.

Feasibility
The number of items/indicators, score range and interpretability. User-friendly was also evaluated.

Specification of an accountable entity
Extend to which the tool determined which organizations or individuals were responsible for reporting and completing the tool.

Strengths and limitations
Characteristics of the tools that were described in the tool and in the literature which can be useful or barriers for their applications in decision making. https://doi.org/10.1371/journal.pone.0272037.t002

Results
Overall, 3714 articles and 67 grey literatures were identified through the searching process. After the screening stage, a total of 34 documents (25 articles and 9 reports) that used 14 toolkits were found. Fig 1. illustrated the searching process. Table 3 provides a list of the features of each toolkit. The Joint External Evaluations (JEE) instrument was the first toolkit developed after the issue of the IHR framework in 2005 [14], following by the Self-Assessment Annual Reporting (SPAR) in 2007. These two toolkits adapted the IHR conceptual frameworks in measuring the risk and vulnerability of each country to epidemic [15]. The most recent instrument was the EpiRisk Tool, which was developed to evaluate the potential severity of an infectious epidemic in a country [18]. Most of the toolkits were developed by the United States organizations (5/14) [12,[26][27][28][29][30], following by European institutions (for example England or European Centre for Disease Prevention and Control) [31][32][33][34] and international organizations (e.g. World Health Organization) [14,15]. The number of criteria/dimensions in each instrument ranged from 2 to 14 with the number of indicators/items ranging from 7 (for Threat and Hazardous Incident Risk Assessment (THIRA) [27] to 140 (for Global health security index) [29]. Levels of setting were usually at a global or national level, while only several scales have been developed specifically to serve the community levels (such as THIRA, US CDC Public Health Preparedness Capabilities and the CDC's Social Vulnerability Index). Notably, only 5/14 instruments had periodical assessments; for example, JEE, SPAR, US CDC Public Health Preparedness Capabilities, and Cambridge Global Risk Index had annual reports while the CDC's Social Vulnerability Index was biannually assessed. Table 4 indicates the breaths of 14 selected measures. In this study, we classified the contents or indicators of selected instruments into three major groups: system components, demographic and community components, and other specific components. Overall, none of the toolkits fully covered all components. The CDC's Social Vulnerability Index [26] covered the least components given that this tool only measured some indicators such as poverty, unemployed, income, education, age, housing or transportation. Meanwhile, the global health security index [17] covered the highest number of components because 140 indicators were used to estimate this index. As for the breadth of measurement, components such as  Table 3. General profiles of risk assessment instruments.

No Instrument
Year of publication  Governance and Legislation, Financing, Health Service Provision, and Human Resources are key domains in almost all toolkits. Meanwhile, we found that some important issues for disease detection and surveillance, such as laboratory or capacity of the community for disease control, were not adequately addressed in several toolkits. Table 5 depicts the usages, properties, strengths and limitations of the selected toolkits. All toolkits determined that they covered the national level except CDC's Social Vulnerability Index. Six of the instruments measured preparedness in epidemic and/or natural disasters (in which epidemic risk assessment was one of the components), and others measured the risk of epidemic. Only the validity of 7/14 instruments was evaluated. Epidemic Preparedness Index toolkit was found to have the strongest correlations with JEE (r = 0.85-0.86 [12,35]) and SPAR (r = 0.62 [12]), following by Global health security index (JEE: r = 0.82 [36]), and INFORM Epidemic Risk Index toolkit (SPAR: r = 0.47 [37] and JEE: r = 0.6 [37]). x x x x x x x x x x x x x x x x x x x 2. Self-Assessment Annual Reporting (SPAR) [15] x 3. CDC' Social Vulnerability Index [26] x x

Discussion
Monitoring the risk and vulnerability of different communities and countries is critically important for national preparedness and responses to the epidemic. This study summarized the profile and developmental history of toolkits, as well as reveals diverse breadths of measurements and their applications in various levels of administration.  [36], or no correlation with COVID-19 mortality rates [43,44] • 54/112 countries had scored lower than the global average [21] • GHSI was positively related to total cases and deaths per million [45]. GHSI had the potential in predicting outcomes of local response.
• GHSI scores of Japan, South Korea, the United Kingdom and the United States were above global and regional averages [51]. GHSI score of Iran was below the global average [51].
• GHSI overestimated the preparedness of OECD countries with high GHSI scores; and underestimated the preparedness of OECD countries with low GHSI score [52] • All Pacific islands countries where data were available belonged to the "least preparedness" group [21].

PLOS ONE
The current paper revealed that fourteen toolkits have been developed to assess the risk of infectious disease epidemics, but most of them lacked validation and were not widely applied. Indeed, before 2020, only seven toolkits were assessed their measurement properties including JEE, SPAR, IDVI, INFORM Epidemic Risk Index, Epidemic Preparedness Index, Global health security index and EpiRisk, raising questions about the applicability of these toolkits in planning response strategies. Prior literature emphasized that although the GHSI was considered a comprehensive evaluation with 140 items, its validity and reliability are questionable, which could lead to miscalculations and misinterpretations [40].
In terms of content, the scales cover quite adequately the aspects related to susceptibility to the disease, but none of the scales can fully cover the whole range of key factors. Most of the instruments use data from a national report or global databases to mirror the policymaking and health care performance. However, contextualized or specific-country factors, which play a critical role in predicting the risk of and response to infectious diseases epidemic [2], are lacking. Lessons learned from the COVID-19 epidemic and other previous outbreaks suggested that the IHR framework has weaknesses in its ability to guide countries to respond to the pandemic [13]. For instance, Kaiser et al. indicated that GHSI had poor predictability and did not meet the need of policymakers in the community [44]. Similarly, Ye Ji et al. found that GHSI was not helpful in assessing the preparedness and response of a country against global pandemic [45]. Although fundamental for developing epidemic risk assessment toolkits, some critical materials are not completely complied by all country members of the World Health Organization for several reasons such as insufficient resources for following recommendations, or unwillingness to change systems following the World Health Organization's notification of outbreaks due to economic losses [13], resulting in the delay in responses to the epidemic. Another example of the weakness of current toolkits is the highest burden of COVID-19 of the United States and European countries, although these nations were among countries with the highest level of health security and epidemic preparedness in all measures [2,28,29]. To date, these countries have been devastated by this epidemic, with the highest number of cases and deaths [1].
It is undeniable that substantial progress among countries has been made by identifying and fulfilling the preparedness gaps via risk assessment toolkits (as shown in previous epidemics such as SARS-CoV, H1N1, Ebola, or Zika) [53]. However, to control the new epidemic as COVID-19, it is necessary to take a holistic view of the available data to identify the current risk, as well as collect more contextualized data (such as individual's knowledge-practice, behaviors, or trust), and proceed to assess risks not only at the national level but also at the community levels [12]. Some efforts have been performed to improve the credibility of the toolkits. For example, Cartaxo et al. integrated indicators of IDVI and contextualized indicators from other databases such as WHO, World Bank and Brazilian Geography and Statistics Institute to enhance the capability in determining the risk of COVID-19 [54]. The authors found a set of 18 indicators with high sensitivity with only 50% of indicators from IDVI [54]. Indeed, the role of core societal behaviors such as networking is critical in disease transmission during the pandemic. Understanding how people perceive the pandemic, how they perform their social behaviors, as well as how socio-environmental factors impact their behaviors are important and should be integrated into the toolkits for effective control [55].
One important observation of pandemic preparedness and control globally was that countries had very different policies and practice. Although the purpose of developing and applying these toolkits was to inform early response at local, national, and global level, the COVID-19 pandemic disproportionately hit every community that suggested the importance of contextual factors to be addressed. In fact, we have found substantial gap in contextual tools and their limited inclusion of contextual factors in the risk assessment tools. Adapting, validating, and routinely improving the validity of these toolkits are important process for diseases control and prevention.
The findings of this study suggested implications for improving the current toolkits as well as proposed new areas for developing new toolkits for COVID-19 control. First, inadequate evidence about the validation of risk assessment toolkits suggests the need for further studies to validate these toolkits to the standard measures such as JEE and SPAR. This information is critically important to ensure that the toolkit can at least reflect the fundamental dimensions of the IHR framework. Second, since limited toolkits have been used for periodical measures, developing mechanisms to facilitate the use of other toolkits in evaluating the change of global, regional, and national preparedness to further epidemics and pandemics, as well as the gaps that should be filled to strengthen the capacity in epidemic responses. Third, further studies are warranted to develop specific domains for each country's priorities, given that the majority of the current toolkits might be relatively broad and not reflect important factors within each country. For example, some important variables for epidemic preparedness, such as community variables (for example, population density, community interactions, or coverage of different protective measures) and individual factors (for instance, knowledge-attitude-practice, social network, or public trust in the government) should be included and measured.
This study had several limitations. First, the searching procedure involved only accessible databases, including Medline/PubMed and Web of Science, and purposively selected sources, such as websites of international organizations, thus, might not fully cover those materials which were i) not written in English, ii) not published or indexed, and iii) internal or local use. However, Medline/PubMed is currently the most comprehensive database for life science and biomedical fields with more than 33 million articles [56], and the Web of Science database contained top-quality papers in more than 21,000 peer-reviewed journals in different disciplines [57]. We also performed searches on the websites of different prestigious public health organizations to identify the current epidemic risk and vulnerability assessment instruments. We believed that with our searching strategy, the searching results were credible in covering most of the necessary articles. Second, we did not search the documents that were published in preprint databases such as Medrxiv because they were not peer-reviewed and could have major alterations after completing their review process. Nonetheless, these databases should be considered in further studies to identify potential valuable toolkits for epidemic risk assessment.

Conclusion
The findings of this study indicated the gaps in evidence on the validity and responsiveness of current risk assessment toolkits as well as the inclusion of population and contextual factors in quantifying epidemic risks. We call for global and national efforts in developing more contextualized and responsive epidemic risk assessment scales incorporating specific-disease and -country factors to inform operational decisions making and strengthen countries' capacities in epidemic responses.
Supporting information S1 Checklist. Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) checklist. (DOCX)